🚀 Tokenizer Performance - abnv · Scour

How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers

arxiv.org·3d

⚡Tokenizer Benchmarks

A Comprehensive Benchmark of Language Models on Unicode and Romanized Sinhala

arxiv.org·4h

⚡Tokenizer Optimization

ANN v3: 200ms p99 query latency over 100 billion vectors

turbopuffer.com·1d·

Discuss: Hacker News

⚡Cache-Aware Algorithms

Harper: Free, Open Source Grammar Checker

writewithharper.com·7h·

Discuss: r/webdev

🔄Incremental Lexing

You Probably Don’t Need a Vector Database for Your RAG — Yet

towardsdatascience.com·1d

Automatic compilation of a Pre-Qin philosophy lexicon via large language models

nature.com·1d

🔤Language Tokenizers

Exploring Text Compression

denvaar.dev·1d

📦Compression Algorithms

How Does ChatGPT Work? A Guide for the Rest of Us

producttalk.org

·19h

🎯Finite Automata

t2x - a CLI tool for AI-first text operations

shruggingface.com·1d

🔄Incremental Lexing

Iterative multi-word anagram solver

boulter.com·1d

🤖Suffix Automata

Making a Language

thunderseethe.dev·11h

🔬programming language theory

A Guide to TF-IDF

pub.towardsai.net

·5h

⚡Tokenizer Benchmarks

How poor chunking increases AI costs and weakens accuracy

blog.logrocket.com·20h

⚡Tokenizer Benchmarks

Explainer: Tree-sitter vs. LSP

lambdaland.org·1d

🌲Tree-sitter

MIT’s new ‘recursive’ framework lets LLMs process 10 million tokens without context rot

venturebeat.com·1d·

Discuss: Hacker News, r/technews

🪜Recursive Descent

Transformer Series #3: Positional Encoding - How Transformers Understand "Time" Without a Clock ⏳

dev.to·19h·

Discuss: DEV

🎰Parsing Machines

PHPStan Now 25~40% Faster For Static Analysis

phoronix.com·22h

🏃Escape Analysis

meta-pytorch/segment-anything-fast: A batched offline inference oriented version of segment-anything

github.com·43m

🗺️Region Inference

I [[musttail]] You About a Tokenizer

neilhenning.dev·2d

Trying Out Claude Code with Ollama

slicervm.com·1d·

Discuss: Hacker News

Loading more...